Data Analytics using Python with JupyterLab

1. On the home page of DPS, add the following stages and connect them as shown below:
   • Data Lake: Snowflake
   • Data Analytics: Python with JupyterLab
   • Data Lake: Snowflake

   

2. Configure the Snowflake nodes. Ensure that JupyterLab has access to the S3 node in the data pipeline. See Setting up S3 access from JupyterLab.
3. Click the Python with JupyterLab node. Click Deploy to deploy Python with JupterLab.

Note:

Each stage of a feature in Data Pipeline Studio (DPS) supports only one active JupyterLab node in the pipeline at a time. If you add a second node to the same stage, the previously added node becomes inactive, and only the most recently added node will function.

1. On the Dev screen, click +Add Instance. Select the required instance. When deploying the Python with JupyterLab node in your data analytics pipeline, the memory requirements can vary depending on the code you implement in the Jupyter Notebook. For optimal performance, the Calibo Accelerate platform recommends using a minimum of 4 GB RAM, 50GB Hard-disk, and 2vCPU.


Note:

Deployment of Python with JupyterLab is currently limited to Docker Container mode in Calibo Accelerate. Kubernetes and OpenShift are not supported for this technology.



2. Click Configure to configure the selected instance before adding the technology.
3. In the Cloud Account and Network Setting side drawer, do the following:

Field	Action/ Description
Name	Provide a logical name for your instance.
Hard Disk Size	Select the desired hard disk size from the list.
Cloud Account	The cloud account on which you configure a cloud instance is auto-populated.
Use Public or Private Subnet	Choose between public and private subnets.
Security Groups	Select one or more security groups. They act as virtual firewalls controlling inbound and outbound traffic for instances hosting your Docker containers.
Security Groups	If you configure settings for an Azure instance, select one or more application security groups. This field is optional.
AMI ID (Optional)	Select or enter the unique ID of the AWS AMI that you want to launch.

4. Publish the pipeline.

When you create a data analytics job, you can add the following type of algorithms:

• Predefined algorithms like Random Forest Classifiers, Support Vector Classifier, Logistic Regression, Linear regression and so on.

• Custom algorithms - See Create Custom Algorithm for a Data Analytics Job

To add an algorithm

1. Click +Algorithm. Provide the following information:

   

   1. Algorithm: Select a predefined algorithm like random Random Forest Classifier, Support Vector Classifier and so on, or Custom Algorithm to be used for data analytics.

   2. Algorithm Name: Provide a unique name for the algorithm.

   3. Column Name: Select the column to run the selected algorithm.

   4. Source: Select a file or table from the source, to which the column belongs on which the algorithm will be run.

   5. Target: Select a file or table from the target in which the output of the algorithm run will be stored. You can either select an existing file or table or create a new one.

      

      Note:

      Ensure that the selected column contains binary data only.

2. Click Add.

3. After you click Add, the Creation Status displays the following types of status depending on whether the data analytics job for the selected algorithm is created or not:

   1. In Progress - when the job is being created.

   2. Success - when the job is created successfully.

   3. Failed - when the job creation fails.

   1. To add multiple algorithms, repeat steps

4. To add multiple algorithms, repeat steps 1 and 2.

5. Enable the Continue on Failure setting for the required algorithms. By turning on the setting, if an algorithm in the sequence fails, the remaining algorithms continue to run.

You can add one or more algorithms to a data analytics job. You can either run each algorithm manually or you can run all the algorithms in a sequence. A Jenkins job is created for each stage of the pipeline with the following folder structure and naming convention:

Product_Release_Feature folder > Stage name folder > run_ml_algorithm folder.

To run the algorithms, choose one of the following options:

• Click Run All to run the complete list of algorithms in the given sequence. You can use the drag handle to change the sequence of the algorithms. If Continue on Failure is enabled, the algorithms continue to run in the specified sequence even if one algorithm fails.

  

• Click Run Selected to run a few algorithms. Select the algorithms that you want to run from the list and then click Run Selected.

  

The Run Status of the pipeline can be one of the following:

1. Initiated - The pipeline run has started.

2. Running - The pipeline is running.

3. Success - The pipeline run is successful.

4. Failed - The pipeline run has failed.

5. Terminated -The run is manually terminated.

6. Not Run - If the previous algorithm fails, the subsequent algorithms go into this status.

If you want to view the output of an algorithm that has run successfully, do the following:

• Click the ellipsis (...) and then click Open Visualization to view the dashboard. View the dashboard generated from the pickle file.